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Abstract 

Human-robot systems are expected to have a central role in future space exploration missions that extend beyond low- 
earth orbit [1], As part of a directed research project funded by NASA’s Human Research Program (HRP), researchers 
at the Johnson Space Center have started to use a variety of techniques, including literature reviews, case studies, 
knowledge capture, field studies, and experiments to understand critical human-robot interaction (HRI) variables for 
current and future systems. Activities accomplished to date include observations of the International Space Station’s 
Special Purpose Dexterous Manipulator (SPDM), Robonaut, and Space Exploration Vehicle (SEV), as well as interviews 
with robotics trainers, robot operators, and developers of gesture interfaces. A survey of methods and metrics used in 
HRI was completed to identify those most applicable to space robotics. These methods and metrics included techniques 
and tools associated with task performance, the quantification of human-robot interactions and communication, usability, 
human workload, and situation awareness. The need for more research in areas such as ‘natural’ interfaces, 
compensations for loss of signal and poor video quality, psycho-physiological feedback, and common HRI testbeds were 
identified. The initial findings from these activities and planned future research are discussed. 

Keywords- end-effectors; human robot interaction; space technology; robots; robot control; telerobotics; teleoperation; user 
interface 


I. INTRODUCTION AND GOALS OF THE CURRENT WORK 

Human-robot interaction (HRI) is a research area that seeks to understand, design for, and evaluate the complex 
relationships among variables that affect the way humans and robots work together to accomplish goals [2]. HRI 
research is critical to improving NASA’s current human-robots systems (e.g., the International Space Station’s 
Special Purpose Dexterous Manipulator, Robonaut 2, and Mars Exploration Rover) as well as future systems, which 
are expected to have central roles in space exploration missions. These future roles may include transporting cargo 
on planetary surfaces; collecting and handling samples (e.g., soil or rocks); grappling other space vehicles, satellites, 
or asteroids; collaborating with extravehicular human crewmembers; assembling and servicing habitats; and 
performing in-vehicle inspections [1,3]. 

The characteristics of space robots are vast and diverse. For example. The Robonaut 2 system was recently 
launched on the STS-133 mission to the International Space Station (ISS) to investigate dexterous manipulation in 
zero gravity. After initial experiments, its purpose is envisioned to offload housekeeping and other chores from 
crewmembers and work side-by-side with them as a teammates in repair and maintenance tasks [4-5]. Operators 
from the ground will control it using a telepresence system with head mounted displays, stereo visual feedback, 
force and tactile feedback gloves, and posture trackers. In contrast, the Mars Exploration Rovers (Spirit and 
Opportunity), which have been operating on Mars since January 2004, are typically commanded once per Martian 
solar day. Operators from Earth send commands telling the rovers where to go (either with or without autonomous 
driving capabilities such as terrain assessment), what images and data to collect, and which way to position the 
robotic arm [6]. Communication between operators and robots is limited by time delays (which can be upwards of 
40 minutes) and constraints on bandwidth. Finally, consider the Special Purpose Dexterous Manipulator (SPDM), a 
robot with two symmetrical seven degree-of-freedom (DOF) arms used to change-out Orbital Replaceable Units 
(ORUs), actuate mechanisms, support extravehicular activities (EVA), and manipulate and inspect payloads on the 
Space Station’s exterior [7]. Although originally designed for on-orbit control, operators on the ground command 
SPDM by following pre-planned procedures, sending discrete commands to position the robot, and monitoring 
camera views and telemetry (Observations of SPDM operation, December 2010). 



From these three space robotic system examples - Robonaut 2, Mars Exploration Rovers, and SPDM - it is 
clear that the nature of the human-robot interaction from one system to the next is very different. Yet, each system 
relies on the appropriate design of tasks, displays, controls, and communication between humans and robots. The 
human is always an integral part of the success or failure of the system. 

A. Research Goals 

Researchers at NASA’s Johnson Space Center, funded under the Human Research Program (HRP), have begun a 
multi-year effort to survey the existing field of space robotics and carry out a series of studies to address gaps in 
research which are applicable to both space robotics and the HRI field in general. The goals of the current work are 
to use a variety of techniques (e.g., literature reviews, knowledge capture, case studies, field studies, interviews, 
observations, and experiments) to understand critical HRI variables and their effects on task performance, task 
efficiency, and user satisfaction. The intention is that results will support the development of HRI requirements, 
standards, and design guidelines. 

II. REVIEW OF HRI METHODOLOGIES, METRICS, AND TOOLS 

The work began with a literature review of methods and metrics in HRI and in related fields such as human- 
computer interaction, aviation, automation, teamwork, and artificial intelligence. Over 350 peer-reviewed journal 
articles and proceedings were reviewed. The goal of this review was to categorize general and space-specific ways 
of studying and measuring HRI. Research gaps and the lack of common methods, metrics, and tools emerged from 
this review and provided impetus for the development of research questions for future studies. A select set of 
findings is presented in this paper. 

A. The Need for Methods and Metrics 

The first finding that emerged from the review was the need for methods and metrics that can answer questions 
about HRI across many applications and systems. Several researchers have commented about the lack of a 
comprehensive set of valid and replicable metrics and the methods to aid in design and evaluation of human-robot 
systems [8-11]. Some have attributed this to the newness of the field and the tendency of robot developers and 
researchers to use metrics that are biased towards a specific application rather than answer basic, but critical, 
research questions. In a workshop of fifty-six leading HRI researchers and robotics developers, the need for HRI 
methods and metrics was identified as one of the major needs for future advancement of the field[12]. Metrics 
identified as critical included those associated with robot autonomy, mission complexity, human-robot coordination 
and collaboration, task performance, sensory processing, human intervention, operator workload, logistics, control 
station attributes, and ease-of-use. User experience, social acceptance, and social impacts are additional critical 
factors that require HRI metric development [11], 

B. Task and Performance Metrics 

A task in HRI is defined as a discrete activity that the human-robot system has to complete within an 
environment via an interface [13]. Tasks involves the human obtaining information about the current state of the 
robot and environment, synthesizing this information in a meaningful way, choosing an action or response to the 
situation, implementing the action, and then evaluating the outcome. Different task metrics provide different 
information about success or failure of the human-robot system during each of these task steps. 

Teleoperation of space robotic arms, robotic urban search and rescue (USAR), and robotic explosive ordnance 
disposal (EOD), are three areas in HRI that have developed a number of objective measures of task performance. 
Measures of task performance for control of space robotic arms have included metrics associated with collision 
avoidance, selection of optimal camera views, and appropriate control inputs. For example [14] investigated task 
performance with alternative camera configurations, by analyzing the location and orientation of the end effector, 
time to capture a simulated node, and time to dock the node. These were used to determine human task performance, 
which included: time from start of trial to first controller input, time to complete the task, percent motion (i.e., 
percent of total time the end effector was moving), percent of multiple DOF inputs, angular DOF movements, and 



docking position offset. Other researchers have defined human task performance for a space robotic arm alignment 
task as control movements that reach limits or singularities, translational (X, Y, Z) and rotational (Pitch, Yaw, Roll) 
accuracy, number of forbidden configurations, control effort (as measured by taking the root mean square value of 
control movements along all axes multiplied by maneuver duration), path efficiency (as measured by taking the root 
mean square value of the distance between current robot position and target multiplied by maneuver duration), sum 
of distances in direction opposite to the target, command velocity, maximum velocity, and number of operator head 
movements as a measure of visual search [15-17]. Although these measures of human task performance allow for 
comparisons among display configurations for different space arms robots, the heavy reliance of measurements from 
hand controller inputs (e.g., angular DOF movements) may make them difficult to apply to robots that use different 
types of control inputs, a problem highlighted later in this paper. 

Researchers in space robotics could also draw lessons from the development of methods and metrics of FIR1 in 
USAR and EOD, who have taken the forefront for developing objective measures of human-robot task performance 
for cross-system comparisons. In particular the development of testbeds in these two disciplines should be noted. A 
testbed, in which human-robot systems perform identical tasks and are evaluated in identical ways, can provide an 
objective means for collecting comparing human-robot task performance. When a testbed involves realistic task 
components, then it can also be used for defining criteria for acceptable performance and validating systems. 
Developing a testbed involves an iterative process of steps including: understanding the operational challenges of 
the tasks and environments, selecting a subset of representative tasks, identifying appropriate metrics and criteria for 
performance, and validating the testbed [18-19]. A testbed needs to addresses the characteristics of the task 
sufficiently, be used in a timely manner, and use meaningful metrics that can affect design decisions. 

Although there have been testbeds created for space robotic applications, particularly for space robotic arms, 
these have been largely focused on developing simulations to validate specific technology or behavioral algorithms 
of robotic systems [20-22] Unlike the testbeds developed for USAR or EOD robots, these testbeds are designed to 
simulate existing robots rather than provide a set of representative tasks and human-robot task performance metrics 
that can be used to compare different robot systems or interfaces. There is a need to develop these types of testbeds 
embedded with unique space-related characteristics as well as developing testbeds for dexterous robots, such as 
Robonaut 2, that work in close proximity to crewmembers. These testbeds for dexterous robots may need to have 
metrics associated with teamwork, communication, and decision making. 

C. Research Gaps 

The literature review on methods and metrics reveals some FIRI interface gaps in space-robotics research 
critical to system usability. Although not inclusive, the match between input device (e.g., gesture, voice, physical 
control) and task, compensations for poor or ambiguous video feedback, and communication issues (i.e., loss of 
signal, communication delay, natural communication) appear to have significant effects on FIRI across a number of 
space applications. Additionally, non-intrusive measures of situation awareness and workload, such as behavioral or 
psycho-physiological measures, need to be validated for use in HRI across space domains. 

Methodological considerations that were identified included robot fidelity for research. Experimental control in 
HRI research can be a challenge with physical robots, as these robots need to act reliably over time and be easy to 
manipulate within the variables of interest. Thus alternative methods to study HRI for space applications need to be 
developed and validated. These methods may included concurrent and retroactive field studies, studies in analog 
environments, “Wizard-of-Oz” techniques, studies with virtual robots, and video trial methods. 

III. KNOWLEDGE CAPTURE OF SPDM OPERATIONS 

Concurrent to the literature review, a survey of current and future space robotics work was completed, along 
with knowledge capture on SPDM operations. During knowledge capture of SPDM, the team reviewed operator 
manuals, training materials, operational procedures, and lesson-learned presentations. The review of these 
documents provided the team with system and operational knowledge that complemented observations of SPDM 
operations and discussions with SPDM operators and trainers. 



The approach uncovered some discrepancies between research in space robotic operations and real operations, 
as well as areas of applicable research. For example, the majority of research on teleoperated space robotic arms 
investigated human control of the arm with continuous input devices, usually a rotational hand controller (RHC) and 
a translational hand controller (THC) [14-16, 23-24]. These studies assumed that a single human operator performs 
real-time decision making for controlling a robot arm. The knowledge capture process, however, revealed that 
operators on the ground control SPDM through discrete inputs (i.e., uplinked commands) that are governed by 
detailed procedures created weeks in advance. Additionally, multiple individuals - the ISS Robotics Officer, System 
Officer, and Task Officer - are involved in the operations, each with specific roles. Teamwork and appropriate 
communication between these individuals is essential to operations and decision-making. For example, two officers 
are required to verify a command before it is sent (i.e., call-and-verify). There are also constraints in operations due 
operational safety requirements, such as all commands need an initial “arm” (select the command), and second 
“arm” (confirm the command), and a “fire” (send the command). Fland controllers would not comply with current 
operational requirements. Within the current operational paradigm, it would be difficult to verify the status of hand 
controller inputs during a loss of signal situation. Our discussions with operators also revealed that, in general, 
sequences of commands or scripts are avoided due to the long verification process required and inability to change 
the script without additional verification. Thus, there is clear need to address FIR1 issues related to current operations 
that can make control more efficient within the historical constraints of NASA activities. 

Observations of SPDM operations (December 2010) also revealed several concerns about interface adequacy. 
One observation was that non-optimal lighting conditions from camera views tended to “wash-out” camera overlays. 
Overlays on end-effector camera views for grappling were also highly dependent on calibration, which could be 
offset by small vehicle vibrations and changes due to mechanical tolerances that are not represented in the modeling 
program. There were no overlays for guiding operator inputs when the target was not in the camera field of view. 
Camera views, in our opinion, appeared ambiguous as to distance between objects and robot speed. The loss of high- 
gain antenna signals (i.e., camera feedback) was another observed issue that affected operations. No motions 
commands were allowed during these periods or immediately preceding these periods, which could last upwards of 
20 minutes. In terms of information, operators were presented with a large amount of telemetry that was not task- 
specific. These, and other observations presented themselves as research opportunities for future work. 


IV. FORWARD WORK 

Building upon the foundational work of the past few months, the FIRI team at NASA’s Johnson Space Center is 
working towards a series of studies and activities to address some of the critical issues identified as gaps in research. 
One of these near-term activities will be to bring together a diverse group of experts in FIRI in a workshop later this 
year. Topics planned for discussion include identification and effects of variables of interest across three classes of 
space robotics - Arm/Heavy Lift, Mobility, and Dexterous. Numerous near- and long-term studies are also planned. 
One set of studies will focus on overlay guidance for operators. An area of interest is the effects of command- 
guidance versus situation-guidance (i.e., providing command input information and errors versus providing natural 
guidance cues) on human-robot task performance across input types and tasks. Input types will include discrete 
commanding similar to current SPDM operations. Other overlay studies may look at superimposed versus integrated 
heads-up information on displays across information type (e.g., information that needs to be continuously monitored 
versus intermittently monitored). A set of studies will be carried out to investigate variables associated with camera 
configurations and display layouts to identify ways to improve operator awareness of a robots status relative to its 
environment during various tasks. Other studies are planned that will focus on developing gesture and voice 
vocabulary for rover-type robots. The ability of individuals to learn and recall traditional and novel control inputs 
will be assessed. Finally, the team plans to conduct studies related to ways operators could regain awareness of robot 
status after a loss of signal event. Throughout these studies, the team plans to apply and refine tools, methods, and 
metrics for measuring HRI. 
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